Skip to content
This repository has been archived by the owner on Feb 7, 2025. It is now read-only.

OpenAI PR Agent Setup #1405

Closed
wants to merge 25 commits into from
Closed

OpenAI PR Agent Setup #1405

wants to merge 25 commits into from

Conversation

pluckyswan
Copy link
Contributor

@pluckyswan pluckyswan commented Oct 9, 2024 •
edited by github-actions bot
Loading

User description

OpenAI PR Agent Setup

  • Added a new PR agent AI reviewer.
  • Our model is using ChatGPT-4 currently as it was recommended as a good text reading LLM. We can always use a different LLM with Azure OpenAI or run different LLMs as a back up.
  • Using mostly default settings, PR Agent is set to Auto-Review/Auto-Improve.
    • We removed auto-describe as we want to encourage engineers to review their stories as part of the PR review process when writing up a PR Description.
  • See https://qodo-merge-docs.qodo.ai/tools/ for all the manual triggering features.
  • See https://qodo-merge-docs.qodo.ai/usage-guide/configuration_options/ for all configuration options.
    • Feel free to look at configurations you think we may need. We use mostly default for now but it can be adjusted accordingly.

Issue

#1357

Checklist

Note: You may remove items that are not applicable

PR Type

enhancement, configuration changes

Description

  • Added a new GitHub Actions workflow to integrate OpenAI's PR Agent, named 'Perry the PR Agent Platypus', which triggers on pull requests and comments.
  • Configured the PR Agent to use Azure's OpenAI API with specific version settings to ensure compatibility.
  • Created a TOML configuration file for the PR Agent, specifying model settings, review behaviors, and other operational settings.
  • The PR Agent is set to automatically review PRs, provide code suggestions, and manage PR descriptions without auto-labeling.

Changes walkthrough 📝

Relevant files
Configuration changes
pr_agent.yml
Setup GitHub Actions Workflow for OpenAI PR Agent               

.github/workflows/pr_agent.yml

  • Added a new GitHub Actions workflow named 'Perry the PR Agent
    Platypus'.
  • Configured triggers for pull requests, pull request review comments,
    and issue comments.
  • Set up environment variables for the PR Agent using GitHub secrets.
  • Specified the use of Azure's OpenAI API with specific version
    settings.
    		•  +29/-0   
    .pr_agent.toml
    Configure PR Agent Settings in TOML File                                 

    .pr_agent.toml

  • Defined configurations for the PR Agent including model settings and
    GitHub action behaviors.
  • Configured PR reviewer settings such as the number of code suggestions
    and inline comments.
  • Set up PR description and code suggestion behaviors.
    		•  +43/-0   

    💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

    @pluckyswan @saquino0827 @jherrflexion
    added initial setup for openai pr agent
    592e947 
    Co-authored-by: saquino0827 <[email protected]>
Co-authored-by: James Herr <[email protected]>
    @github-actions GitHub Actions
    Copy link

    github-actions bot commented Oct 9, 2024

    Failed to generate code suggestions for PR

    1 similar comment
    @github-actions GitHub Actions
    Copy link

    github-actions bot commented Oct 9, 2024

    Failed to generate code suggestions for PR

    @pluckyswan pluckyswan marked this pull request as ready for review October 9, 2024 23:00
    @github-actions GitHub Actions
    Copy link

    github-actions bot commented Oct 9, 2024

    Failed to generate code suggestions for PR

    @saquino0827
    Copy link
    Contributor

    /describe

    @saquino0827 @somesylvie
    @pluckyswan @halprin
    Adding every configuration combination possible
    595737d 
    Co-authored-by: Sylvie <[email protected]>
Co-authored-by: pluckyswan <[email protected]>
Co-authored-by: halprin <[email protected]>
    @saquino0827
    Copy link
    Contributor

    @CodiumAI-Agent /review

    @CodiumAI-Agent
    Copy link

    CodiumAI-Agent commented Oct 10, 2024 •
    edited by github-actions bot
    Loading

    PR Reviewer Guide 🔍

    (Review updated until commit 94d1509)

    Here are some key observations to aid the review process:

    🎫 Ticket compliance analysis 🔶

    1357 - Partially compliant

    Fully compliant requirements:

    • Implement an AI PR review system using a third-party library or service.
    • Remove auto-labelling from the PR Agent.

    Not compliant requirements:

    • Remove the trigger for pull_request_review_comment.
    • Test the PR Agent with example PRs in Trusted Intermediary and SFTP Ingestion Service.
    • Create new cards for migration of Azure LLM hosting and setup of SFTP Ingestion Service with PR Agent.
    ⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
    🧪 No relevant tests
    🔒 No security concerns identified
    ⚡ Recommended focus areas for review

    Configuration Issue
    The trigger for pull_request_review_comment is still present, which should have been removed according to the ticket requirements.

    @saquino0827 saquino0827 reopened this Oct 10, 2024
    @CodiumAI-Agent
    Copy link

    Persistent review updated to latest commit 595737d

    @github-actions GitHub Actions
    Copy link

    Failed to generate code suggestions for PR

    @CodiumAI-Agent
    Copy link

    Persistent review updated to latest commit 595737d

    @saquino0827 saquino0827 reopened this Oct 10, 2024
    @github-actions GitHub Actions
    Copy link

    Failed to generate code suggestions for PR

    @github-actions GitHub Actions
    Copy link

    PR Description updated to latest commit (4448436)

    @somesylvie
    Copy link
    Contributor

    /review

    @github-actions GitHub Actions
    Copy link

    github-actions bot commented Oct 15, 2024 •
    edited
    Loading

    PR Code Suggestions ✨

    Latest suggestions up to 94d1509

    Explore these optional code suggestions:

    CategorySuggestion                                                                                                                                    Score
    Possible issue
    Correct the syntax of environment variable keys in the GitHub Actions YAML file

    Ensure that the environment variable keys in the GitHub Actions YAML file follow the
    correct syntax. Replace the incorrect dot notation with underscores.

    .github/workflows/pr_agent.yml [25-27]

    -OPENAI.API_TYPE: 'azure'
-OPENAI.API_BASE: ${{ secrets.AZURE_OPENAI_BASE_URL }}
-OPENAI.DEPLOYMENT_ID: ${{ secrets.AZURE_OPENAI_DEPLOYMENT_ID }}
+OPENAI_API_TYPE: 'azure'
+OPENAI_API_BASE: ${{ secrets.AZURE_OPENAI_BASE_URL }}
+OPENAI_DEPLOYMENT_ID: ${{ secrets.AZURE_OPENAI_DEPLOYMENT_ID }}
    Suggestion importance[1-10]: 10

    Why: The suggestion correctly identifies and fixes a syntax error in the environment variable keys, changing from dot notation to underscores. This is crucial as the dot notation would not be correctly recognized in the GitHub Actions environment, potentially causing the workflow to fail or behave unexpectedly.

    		10

    Previous suggestions

    Suggestions up to commit c9f1a28
    CategorySuggestion                                                                                                                                    Score
    Best practice
    Standardize environment variable naming convention to use underscores

    Ensure that the environment variable names follow a consistent format. The current
    format mixes dots and underscores which might lead to confusion or errors in usage.

    .github/workflows/pr_agent.yml [25-27]

    -OPENAI.API_TYPE: 'azure'
-OPENAI.API_BASE: ${{ secrets.AZURE_OPENAI_BASE_URL }}
-OPENAI.DEPLOYMENT_ID: ${{ secrets.AZURE_OPENAI_DEPLOYMENT_ID }}
+OPENAI_API_TYPE: 'azure'
+OPENAI_API_BASE: ${{ secrets.AZURE_OPENAI_BASE_URL }}
+OPENAI_DEPLOYMENT_ID: ${{ secrets.AZURE_OPENAI_DEPLOYMENT_ID }}
    Suggestion importance[1-10]: 8

    Why: The suggestion correctly identifies an inconsistency in the environment variable naming convention and provides a standardized format using underscores, which improves readability and reduces potential errors in usage.

    		8
    Maintainability
    Update configuration setting descriptions to accurately reflect defaults and enhance clarity

    Correct the default values and descriptions for configuration settings to accurately
    reflect the actual defaults and enhance clarity.

    .pr_agent.toml [18-24]

    -inline_code_comments = true # If set to true, the tool will publish the code suggestions as comments on the code diff. Default is false.
-enable_review_labels_security=false # If set to true, the tool will publish a 'possible security issue' label if it detects a security issue. Default is true.
+inline_code_comments = true # Default is true. If set to true, the tool will publish the code suggestions as comments on the code diff.
+enable_review_labels_security=false # Default is false. If set to true, the tool will publish a 'possible security issue' label if it detects a security issue.
    Suggestion importance[1-10]: 7

    Why: Correcting default values and descriptions in the configuration improves clarity and ensures that the documentation matches the actual behavior, which is crucial for maintainability and user understanding.

    		7
    Enhancement
    Simplify model configuration settings

    Remove redundant model configurations if they are not necessary, as it simplifies
    configuration management and reduces potential errors.

    .pr_agent.toml [2-4]

     model="gpt-4"
-model_turbo="gpt-4"
 fallback_models=["gpt-4"]
    Suggestion importance[1-10]: 6

    Why: Removing redundant model configurations simplifies the configuration file and reduces potential errors, making the suggestion valuable for maintainability.

    		6
    Possible issue
    Confirm API version compatibility with deployed services

    Verify the compatibility of the specified API versions with the deployed services to
    avoid potential integration issues.

    .github/workflows/pr_agent.yml [28-29]

    -OPENAI_API_VERSION: '2023-03-15'
-AZURE_API_VERSION: '2023-03-15-preview'
+OPENAI_API_VERSION: '2023-03-15' # Ensure this version is supported by the deployed OpenAI service
+AZURE_API_VERSION: '2023-03-15-preview' # Ensure this version is supported by the deployed Azure service
    Suggestion importance[1-10]: 5

    Why: This suggestion is a precautionary measure to verify API version compatibility. It's a good practice but not a direct improvement or bug fix, hence the moderate score.

    		5
    Suggestions up to commit 4448436
    CategorySuggestion                                                                                                                                    Score
    Possible issue
    Replace periods in environment variable keys with underscores to ensure compatibility and prevent issues

    Ensure that the environment variable keys do not contain periods as it might not be
    supported or could lead to unexpected behavior. Replace OPENAI.API_TYPE,
    OPENAI.API_BASE, and OPENAI.DEPLOYMENT_ID with underscore-separated keys.

    .github/workflows/pr_agent.yml [25-27]

    -OPENAI.API_TYPE: 'azure'
-OPENAI.API_BASE: ${{ secrets.AZURE_OPENAI_BASE_URL }}
-OPENAI.DEPLOYMENT_ID: ${{ secrets.AZURE_OPENAI_DEPLOYMENT_ID }}
+OPENAI_API_TYPE: 'azure'
+OPENAI_API_BASE: ${{ secrets.AZURE_OPENAI_BASE_URL }}
+OPENAI_DEPLOYMENT_ID: ${{ secrets.AZURE_OPENAI_DEPLOYMENT_ID }}
    Suggestion importance[1-10]: 9

    Why: Using periods in environment variable keys can lead to compatibility issues or unexpected behavior in certain systems. The suggestion correctly identifies and fixes this potential issue by replacing periods with underscores, which is a best practice in environment variable naming.

    		9
    Align the configuration setting with the documented default for security label publishing

    Correct the inconsistency in the configuration for enable_review_labels_security.
    The comment suggests the default is true, but it is set to false. Align the setting
    with the intended default value or adjust the comment accordingly.

    .pr_agent.toml [29]

    -enable_review_labels_security=false # If set to true, the tool will publish a 'possible security issue' label if it detects a security issue. Default is true.
+enable_review_labels_security=true # If set to true, the tool will publish a 'possible security issue' label if it detects a security issue. Default is true.
    Suggestion importance[1-10]: 8

    Why: The suggestion correctly identifies a discrepancy between the configuration setting and its documentation comment regarding security label publishing. Aligning these ensures clarity and consistency, which is crucial for maintaining accurate documentation and expected behavior.

    		8
    Suggestions up to commit 4f5f36a
    CategorySuggestion                                                                                                                                    Score
    Possible issue
    Correct the environment variable keys to use underscores instead of dots

    Ensure that the environment variable keys in the GitHub Actions YAML file use
    correct formatting. Replace the incorrect dot notation with underscores.

    .github/workflows/pr_agent.yml [22-24]

    -OPENAI.API_TYPE: 'azure'
-OPENAI.API_BASE: ${{ secrets.AZURE_OPENAI_BASE_URL }}
-OPENAI.DEPLOYMENT_ID: ${{ secrets.AZURE_OPENAI_DEPLOYMENT_ID }}
+OPENAI_API_TYPE: 'azure'
+OPENAI_API_BASE: ${{ secrets.AZURE_OPENAI_BASE_URL }}
+OPENAI_DEPLOYMENT_ID: ${{ secrets.AZURE_OPENAI_DEPLOYMENT_ID }}
    Suggestion importance[1-10]: 9

    Why: The suggestion correctly identifies a syntax error in the environment variable keys within the GitHub Actions YAML file. Using dots instead of underscores could lead to misinterpretation or errors during the parsing of these variables, potentially causing the workflow to fail. This correction is crucial for the proper functioning of the GitHub Actions workflow.

    		9
    Suggestions up to commit add342d
    CategorySuggestion                                                                                                                                    Score
    Best practice
    Modify environment variable names to use underscores for better compatibility and standardization

    Ensure that the environment variable names follow the standard convention of using
    underscores instead of dots for separation to avoid potential parsing issues.

    .github/workflows/pr_agent.yml [22-24]

    -OPENAI.API_TYPE: 'azure'
-OPENAI.API_BASE: ${{ secrets.AZURE_OPENAI_BASE_URL }}
-OPENAI.DEPLOYMENT_ID: ${{ secrets.AZURE_OPENAI_DEPLOYMENT_ID }}
+OPENAI_API_TYPE: 'azure'
+OPENAI_API_BASE: ${{ secrets.AZURE_OPENAI_BASE_URL }}
+OPENAI_DEPLOYMENT_ID: ${{ secrets.AZURE_OPENAI_DEPLOYMENT_ID }}
    Suggestion importance[1-10]: 9

    Why: The suggestion correctly identifies a common best practice in environment variable naming, which enhances compatibility and avoids parsing issues. This is a critical improvement for ensuring the workflow functions correctly.

    		9
    Performance
    Refine the trigger condition for 'issue_comment' to enhance workflow efficiency

    Use a more specific trigger condition for the 'issue_comment' event to avoid
    unnecessary workflow runs.

    .github/workflows/pr_agent.yml [5]

     issue_comment:
+  types: [created]
    Suggestion importance[1-10]: 7

    Why: Specifying the type of 'issue_comment' events can prevent unnecessary workflow triggers, thus saving resources and improving efficiency. This is a relevant and beneficial enhancement.

    		7
    Enhancement
    Enable the generation of code suggestions in PR reviews by adjusting the configuration

    Set 'num_code_suggestions' to a positive value to enable the generation of code
    suggestions during PR reviews.

    .pr_agent.toml [21]

    -num_code_suggestions=0
+num_code_suggestions=3
    Suggestion importance[1-10]: 6

    Why: Setting 'num_code_suggestions' to a positive value activates a useful feature for automated code suggestions during PR reviews, which can significantly enhance the review process.

    		6
    Maintainability
    Simplify the configuration by removing redundant model specifications

    Remove the redundant model specification for 'model_turbo' if it is intended to be
    the same as 'model'.

    .pr_agent.toml [2-3]

     model="gpt-4" # the OpenAI model you've deployed on Azure (e.g. gpt-3.5-turbo)
-model_turbo="gpt-4" # the OpenAI model you've deployed on Azure (e.g. gpt-3.5-turbo)
    Suggestion importance[1-10]: 4

    Why: Removing redundant entries in configuration files can improve maintainability and clarity. However, this change is relatively minor and does not impact functionality directly.

    		4

    @somesylvie somesylvie closed this Oct 15, 2024
    @somesylvie somesylvie reopened this Oct 15, 2024
    @github-actions GitHub Actions
    Copy link

    Persistent review updated to latest commit c9f1a28

    somesylvie and others added 2 commits October 15, 2024 15:35
    @somesylvie @saquino0827
    adjust some more settings
    c0e5dba 
    Co-Authored-By: Samuel Aquino <[email protected]>
    @somesylvie @saquino0827
    one more
    94d1509 
    Co-Authored-By: Samuel Aquino <[email protected]>
    @github-actions GitHub Actions
    Copy link

    Persistent review updated to latest commit 94d1509

    @somesylvie somesylvie closed this Oct 15, 2024
    @somesylvie somesylvie reopened this Oct 15, 2024
    @github-actions GitHub Actions
    Copy link

    Persistent review updated to latest commit 94d1509

    @sonarqubecloud SonarQubeCloud
    Copy link

    Quality Gate Passed Quality Gate passed

    Issues
    0 New issues
    0 Accepted issues

    Measures
    0 Security Hotspots
    0.0% Coverage on New Code
    0.0% Duplication on New Code

    See analysis details on SonarCloud

    @somesylvie somesylvie closed this Oct 15, 2024
    Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
    Reviewers

    @basiliskus basiliskus basiliskus left review comments

    @saquino0827 saquino0827 saquino0827 left review comments

    @somesylvie somesylvie somesylvie left review comments

    Assignees
    No one assigned
    Labels
    configuration changes enhancement Review effort [1-5]: 3
    Projects
    None yet
    Milestone
    No milestone
    Development

    Successfully merging this pull request may close these issues.
    5 participants
    @pluckyswan @saquino0827 @CodiumAI-Agent @somesylvie @basiliskus